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Abstract — We study the moderate-deviations (MD) setting for 
lossy source coding of stationary memoryless sources. More 
specifically, we derive fundamental compression limits of source 
codes whose rates are R{D) ± e„, where R{D) is the rate- 
distortion function and is a sequence that dominates ^/TJn. 
This MD setting is complementary to the large-deviations and 
central limit settings and was studied by Altug and Wagner for 
the channel coding setting. We show, for finite alphabet and 
Gaussian sources, that as in the central limit-type results, the 
so-called dispersion for lossy source coding plays a fundamental 
role in the MD setting for the lossy source coding problem. 

Index Terms — Moderate-deviations, rate-distortion, dispersion. 



I. Introduction 

Rate-distortion theory (T) consists in finding the optimal 
compression rate for a source X ^ P subject to the condition 
that there exists a code which can reproduce the source to 
within a distortion level D. The optimal compression rate for 
the distortion level D is known as the rate-distortion function 
R{P,D). This function can be expressed as the minimization 
of mutual information over test channels |T|. 

It is also of interest to study the excess distortion probability 
for codes at rate R > R{P,D). This is the probability 
that the average distortion between X" and its reconstruction 
X" exceeds D. The exact exponential rate of decay of this 
probabihty was derived by Marton 121 for discrete memoryless 
sources (DMSs). This was extended to Gaussian f3l and 
general sources [4}. These results belong to the theory of large- 
deviations (LD) and are reviewed in Section [III 

With the revival of interest in second-order coding rates 
and dispersion analysis HJ-lTl, various researchers have also 
studied the fundamental limit of lossy compression subject 
to the condition that the probability of excess distortion is 
no larger than e > 0. In particular, it was shown in |i8j and 
independendy in S, |[lOl that 



R{n,D,e) w i?(P, £>) 



(1) 



where R{n, D, e) is the optimal rate of compression of a 
memoryless source at blocklength n and V{P, D) is known 
as the dispersion of the source. Eq. ([T]) holds true for both 
discrete and Gaussian sources and belongs to the realm of 
central limit theorem (CLT)-style results. 

In this paper, we operate in a moderate-deviations (MD) 
regime [Tl , Section 3.7] that "interpolates between" the LD 



and CLT regimes. In particular, we study the performance of 
source codes of rates i?„ = R{P, D) ± where e„ is a 
sequence that is asymptotically larger than \f^Jn (cf. ([TJ)- 
Our results apply to both finite alphabet and Gaussian sources 
but do not reduce to the LD or CLT settings. Moreover, neither 
the LD nor CLT results specialize to our setting. We show that 
the dispersion V(P^D) also plays a fundamental role in this 
MD setting. Besides studying the excess distortion probability, 
we also study the complementary probability (also termed the 
probability of correct decoding) for codes whose rates are 
below the rate-distortion function. Similarly, the fundamental 
nature of the dispersion is revealed. 

This work is inspired by the work on MD in the context 
of channel coding 112], fOl. It was shown in lfT2]| that for 
positive discrete memoryless channels (i.e., Vl^(2/|a;) > for 
all x^y), the dispersion also governs the "MD exponent" 

lim ^ log e(/„, y>„, ly) ^ - . ■ (2) 

ri^oc ne^ 2V{W) 

The direct part was proved by considering the Taylor ex- 
pansion of Gallager's random coding exponent. We also use 
this proof strategy. In |13|, several assumptions in fT2| were 
relaxed and the relations between the MD and CLT were 
clarified. Concurrent to this work, Sason lfT4l studied MD 
for binary hypothesis testing. Finally, we mention that He et 
al. il5J studied the redundancy of the Slepian-Wolf problem 
which is also related to lISl- lfTOl and to the current problem. 

II. System Model and Basic Definitions 

Let 'P{X)he the set of probability mass functions supported 
on the finite alphabet X. Let VniX) C V{X) be the set of n- 
types. For a type Q £ Vn{X), let Tq be the set of sequences 
x" of type Q, i.e., the type class. The reproduction alphabet 
is denoted as X. In addition, \el d : X y. X ^ R+ be a 
distortion measure such that for every x £ X, there exists an 
io e X for which d{x, xo) = 0. The average distortion is 
:= ^ X]r=i '^(^j' ^0- For ^ function f : A ^ B, 
the notation ||/|| := |/(-4)| denotes the cardinality of its range. 

A DMS X" - n"^^ P{x,) is described at rate R by an 
encoder. The decoder receives the description index over a 
noiseless link and generates a reconstruction sequence X" G 
X". We now remind the reader of the rate-distortion problem. 

Deflnition 1. A rate-distortion code consists of(i) an encoder 
fn '■ — > M.n and (ii) a decoder (pn : Ain X"- The rate 



of the code is Rn :— i log 

The rate-distortion function B.{P, D) is defined as the 
infimum of all numbers R for which there exists codes 
{(/„, (p„)}„gN for which the probability of excess distortion 

e{f^,^n,P,D) :-P(d(X",^„(/„(X"))) > D) (3) 

is arbitrarily small for sufficiently large blocklengths n. The 
rate-distortion function |[T| can be expressed as 



R{P,D)^ min I{P,W), 

W:K[d{X,X)]<D 



(4) 



where E[d{X,X)] 'Ex,x Pi^W{S:\x)d{x,x). Another 
fundamental quantity introduced by Ingber and Kochman [SJ 
is the dispersion far lossy source coding 



ViP,D) ■.= yarx[R'iX;P,D)], 



(5) 



where R'{x- P, D) = -q^^R{P, D) for x e A" is the partial 
derivative of the rate-distortion function w.r.t. P{x) (assuming 
it exists). In Q, the variance is taken w.rt. the distribution P 
and R'{X;P,D) is a function of the random variable X. In 
fact, the term dispersion is usually an operational one but since 
it was shown in [8| that the operational defintion coincides 
with the one in (|5]l, we will abuse terminology and use the 
generic term dispersion for both quantities. 

We analyze e(/„, P, £>) in the so-called MD regime 
where the rate of the code i?„ := ^ log ||/„|| — R{P, D) + e„ 
for some sequence e„. Clearly, if e„ — > 0, then _R„ -^• 
R{P,D). When the rate of the code i? is a constant strictly 
above R{P, D), Marton showed that 



lim -\oge{fn,fn,P:D)^- 

n— >oo TL 

where Marton 's exponent is defined as 



-F{P,R,D), 



F(P,R,D):^ min D(Q\\P). 

QeV(X):R(Q,D)>R 



(6) 



(7) 



The exponent is positive for R > R{P, D). One can also con- 
sider the probability of correct decoding 1 — e(/„, P, D). 
In ||T6l pp. 156], it was shown that: 

lim - log (1 - e(/„, P, D)) - -G(P, R, D), (8) 



where the exponent for correct decoding is 



GiP,R,D) 



min D(Q\\P). 

2eV(X):R(Q.,D)<R 



(9) 



The exponent is positive for R < R{P,D). These limits 
and exponents are Sanov-like LD results [11]. We present 
MD versions of Marton's and Iriyama's results where the 
normalizations in (|6]l and ([8]) need not be i. 

III. Discrete Memoryless Sources (DMS) 
Our main result for a DMS with bounded distortion measure 
(i.e. d : A" X A" — > [0,(imax]) is stated as follows: 

Theorem 1. Let e„ be any positive sequence satisfying 

2 

Tlf 

0, lim = oo. (10) 

n-i-oo logn 



lim e„ 

n— )-oo 



That is, e„ = n o(l). Assume that R{Q,D) 

is twice differentiable w.r.t. Q in a neighborhood of P 
and V{P,D) > 0. There exists a rate-distortion code 
{(/«,</5n)}nGN with rates ilog||/„|| < R{P, D) + e„ such 
that 

1 1 

lim sup — 2" log e(/„, P, D) < 



net, 



'2V{P,D)' ^^^^ 

Furthermore, every rate-distortion code {(/«, 'y3n)}neN with 
rates ^ log < P(P, D) + e„ must satisfy 



1 



lim inf — j log <Pri, P,F>)> 



1 



(12) 



n^oo ne^ - - ■ 2V{P,D)' 

Though somewhat ungainly, the log factor in ( fTOb appears to 
be essential because the proof hinges on the method of types. 
So our analysis does not completely close the gap between 
the CLT and LD regimes. This log factor is unnecessary in 
the Gaussian case as will be seen in Theorems |5] and |6] 
Theorem [T] means that if the dispersion V{P, D) is small, 
the "MD exponent" {2V{P,D))^^ is large, corresponding 
to a faster decay in the excess distortion probability. This 
has the same interpretation as in the CLT regime ([TJ. As an 
example, for the Bernoulli source with Hamming distortion, 
the dispersion can be computed as 

'1 - 



V(Bein{a), D) = ail - a) log-" 



(13) 



The parameter that maximizes (resp. minimizes) V{P, D) is 
a « 0.0832 (resp. a = 0,0.5). Thus, the "MD exponent" is 
maximized when the source is deterministic or has maximum 
entropy. The proof uses the following lemma, whose proof 
is essentially identical to that of II17I Theorem 8], where the 
divergence and the constraint set in (|7]l are approximated by 
a quadratic and an affine subspace respectively. 

Lemma 2. If the limit exists, Marton 's exponent satisfies 
^^FiP,RiP,D)+S,D)^ 1 
s^o 5^ 2V{P,D) ^ ' 

In the sequel, we assume that the limit in (fT4l i exists. 
Otherwise, the results are modified accordingly by considering 
the upper and lower limits in (fT4l) and replacing the dispersion 
by its upper and lower limit versions. We first prove the direct 
part of Theorem [U in ( fTTI ) followed by the converse in ( fT2] i. 

Proof: The code construction proceeds along the lines of 
that in [8J. Fix a sequence e„ satisfying JTOl i. From the refined 
type covering lemma by Berger (stated in iflSl ). for every type 
Q G Vni'^) there exists a set Cq that completely D-covers 
Tq (i.e., for every a;" S Tq there exists an G Cq such that 
£")<£)) and Cq has rate 



1 



\og\CQ\ < R{Q,D) + J{\Xl\X\) 



logn 



(15) 



n n 
where J is some function of the size of the alphabets. Consider 
the set C that that is the union of all sets that _D-cover the types 
Q e Un{D, e„), defined as 

UaiD, e„) := {Q eP„(A') : R{Q, D) < R{P, D) + e'„, 

||Q-P||i <e„/ynfVD)}. (16) 



where e',, e„ - J{\X\, \X\Y^ - \X\'-2Sill±ll . The second 
constraint on the li distance of the type Q to the true 
distribution P is to ensure that R{- ,D) is differentiable. This 
is also done in ifTSl Theorem 4]. Note that if e„ satisfies (fTOl l 
so does e' . Now, consider the size of C: 



\C\ = 



E 



\Cq\ 



'2£Vn(X):B.(Q,D)<R(P,D)+€'^ 



< (n + 1)1-^1 exp 

< exp [n {R{P, D) 



R{Q\D) 



j{\xi\x\ 



logri 



(17) 



The first inequality applies (fTsT i and the type counting lemma. 
Furthermore, Q* is the dominating type. The second inequality 
applies the definitions of Un and e^^ . Take /„ to be the function 
that maps a sequence x" e Af" with type P^:" to a predefined 
index in C = UqeUn^Q and take ipn to be the function that 
maps the index to the reproduction sequence in Cp^„ that D- 
covers x". Now, we evaluate the error probability, which is 
the ^"-probability of the types not in Un{D, e„). Consider, 



'{R{Px'^,D)>R{P,D)+e'J 



E 

QeV„(X):RiQ,D}>R(P,D)+e'„ 



< Yl eM-nDiQ\\P)) 

QeP„{X):R{Q,D)>R{P,D)+e'^ 

<{n + 1)1-^1 exp[-nF(P, R{P, D) + e^, D% (18) 

where we applied the type counting lemma and the definition 
of Marton's exponent in the last line. Next, from |19|, 

P(||Px" -P\\i> e.a/Vv) < 21-^1 exp [-nel/i2V)] . (19) 

Combining ( fTSl l and ( fT9] l with the union bound, 

e{fn, (fin, P, D) 

|A'|log(n + l) 



< 2 exp 



2ViP,D) 

where we invoked Lemma |2] with e^^ — o(l) in the role of 5. 
Now, we take the logarithm and normalize by ne^ to assert the 
achievability part of the theorem in ( fTTT i. Note that we used 
the fact that ^ ^ 0. 

Now for the converse, we fix a code {(/«, (pn)}neti of rate 
Rn = ^ log ||/„|| < R{P, D) + e„ and observe that 

eiU,ipn,P,D) > P(d(X",l") > D\£^JV{£^J. (20) 

where the event {R{Px^,D) > i?„ + and Px- 

is the type of X". From the converse of the type covering 
lemma |20, Lemma 3], for any type Q S Vn{X) such that 
i?((5, D) > R, the fraction of Tq that is covered by any set 
is no greater than exp[-n(i?(Q, D) -R + K{\X\, | A'D^^)]. 
Hence, the first term above can be bounded as 



> 1 — exp 



K{\X\,\X\) 



\ogn 



(21) 



Put ;= iK{\X\, \X\) + lY-^. Then, (EB yields 

P(d(X",X") > Dl^^-J > 1 - i > i. (22) 



Hence, it remains to bound the second term in ( [20l ). Let := 
En + ^'n and consider, 

P'^'{£^J^P{R{Px.,D)-R{P,D) > R- R{P,D) + 'i/n) 
> P(i?(Px" , D) - R{P, D) > e„ + 

E P'^i^Q^ 



Q<£P„{X):R{Q,D)>R{P,D)+e'^ 

> ^ exp(-ni?(g||P)) 

QeP„{X):R{Q,D)>R{P,D)+e',^ 



(n + 1)1^1 



>{n + exp [-nD(g(") || P) 



(23) 



where the first inequality is from the definition of P„ < 
R{P, D) + e„ and in the last inequality we defined the type 

g(") := &TgT[).inQ^T:,^f^x):R(Q^D)>R(P^D)+e'^ D{Q II P)- I" the 

appendix, we prove the following key continuity statement. 
Lemma 3. If e'^^ satisfies (fTOl l, the types Q'"' satisfy 
D{Q(^^ IIP) 



lim 



1. 



(24) 



P(P,P(P,i?)+e;„i?) 

Let 7] > 0. For n large enough, the ratio in (|24] | is smaller 
than 1 + ??. Uniting (EOll - (EHi yields 

e(/„,93„,P,i:)) 

> + 1)^1^1 exp[-n(l + 77)P(P, P(P, Z?) + ej,, Z?)] 

^'2 



1 



> 7:(" + l) 



-1^1 



exp 



\2V{P,D) 

The last inequality is an application of Lemma |2] with — 
o(l) in the role of 5. Now, we take the logarithm and normalize 
by ne^ to establish the converse noting that rj is arbitrary, 
= 0(12811) and i^g^ 0. The latter allows us to assert 
that e^/e„ -J' 1. ■ 
Note that the multiplicative nature of (l24l i is necessary to 
establish Theorem[T] The analysis for the probability of correct 
decoding 1 — e(/„, P, D) in the MD regime is analogous 
and is stated in the following: 

Theorem 4. Let e„ be any positive sequence satisfying dlOl ). 
Assume that R{Q, D) is twice differentiable w.r.t. Q in a neigh- 
borhood of P and V{P, D) > There exists a rate-distortion 
code {ifn,'Pn)}n&n with rates ilog||/„|| > R{P,D) - Cn 
such that 



liminf — log (1 - e(/„, (^„, P, D)) > 



1 



n->-oo net, 



2V{P, D) 



(25) 



Furthermore, every rate-distortion code {{fmVn)}n&i with 
rates ^ log ||/n|| > P(P, D) — e„ must satisfy 



lim sup — 2" log (1 ^ ^Un, ^n,P, D)) < - 
Proof: Similar to Theorem [T] 



1 



2V{P, D) 



(26) 



IV. Quadratic Gaussian Source Coding 

We now turn our attention to the quadratic Gaussian setting 
where X" is a length-n vector whose entries are identically 
distributed as zero-mean Gaussians with variance . The 
distortion measure is d{x,x) := (x — i)^. It is known fT) 
that in this case, the rate-distortion function takes the form 



R{a^,D) 



1 , [ ^ 

— loff max < 1 , — 

2 ^ I'D 



(27) 



Furthermore, Ihara and Kubo O showed that the analogue of 
Marlon's exponent in (|7|i also holds in the Gaussian setting. 
Indeed, it is shown that the excess distortion exponent is 



D 



^e2« - 1 



log ( 



(28) 



whenever R > R{a^,D) and zero otherwise. The exponent for 
correct decoding G{a^, R, D) takes the same form as in (|28] | 
when R < R{a^,D) and zero otherwise. In this case, it 
is easy to show by direct differentiation of F{a^ , R, D) (or 
G{a^, R, D)) that the dispersion for lossy source coding is 

1 



V{cJ,D) 



2' 



(29) 



for all cr^ and all D. In analogy to Theorem [T] we have the 
following in the quadratic Gaussian setting: 



Theorem 5. Let e„ be any positive sequence satisfying 



lim e„ = 0, 



lim 



(30) 



There exists a rate-distortion code {(/n, V5n)}neN with rates 
^ log < R{a^,D) + e„ such that 

limsup^loge(/„,(^„,cr^,L') < -1. (31) 



Furthermore, every rate-distortion code {{fn,^n)}n&i with 
rates ^ log ||/n|| < R{P, D) + e„ must satisfy 

liminf ^loge(/„,(/3„,cr^,i:') > -1. (32) 



In contrast to the DMS case, the dispersion for the quadratic 
Gaussian case ( |29l ) is constant. Hence, the exponents in ( l3Tl i 
and ( [32] i are also constant. Also note from ( [30] l that the 
requirement on e„ is less stringent than in the DMS case ( fTOl i. 
In particular, the log factor is no longer required. This is 
because the method of types is not used in the proof. 

Proof: Fix the sequence e„. For the direct part, let us 
consider the set of "empirical variances" 



Un{D, e„) : D) - R{<j\D)\ < e^} 

logs 



(33) 



where := e„ - - By using the definition of 

R{a^,D) in ( |27] i, it is easy to see that e U„ if and only 
if e~^'^" < a'^/a'^ < e^^". We now use a result by Verger- 
Gaugry 11211 Theorem 1.2], which in our context, says that 
6n''^/'^((T^e^'^"/D)"/^ reconstruction points suffice to _D-cover 
length-n vectors x" whose empirical variance i X)i^f ^ ^n- 
Hence, the size of the code is bounded as 

\C\ < 6n5/2(a2e2</D)"/2 < exp{n{R{a^ , D) + e„)), (34) 



where we used the definition of ej^. Hence, the rate i?„ < 
R{(T^,D) + e„ as required. For the probability of excess 
distortion, we have 



<4cxp [-|(e2<-1^2.:, 



2-2€ 



(35) 



The first inequality is by the definition of Un and the union 
bound. The second is an application of the upper bound of 
Cramer's theorem ifTTI applied to the Xi -random variables 
xf I a^. Now note from Taylor's theorem that e^^" — 1 — 2eJ^ = 
2eJ^ + o(£n )■ Taking the logarithm, normalizing by ne^j and 
taking the upper limit of ( [35] l yields the desired result in ( [3T1 ). 

We now turn our attention to the converse. The gist of the 
proof follows from the converse in [3J but, as we shall see, 
the error probability analysis is more intricate. Fix codes of 
rates i?„ = ilog||/„|| < R{a'^,D) + e„. Let the repro- 
duction sequences be denoted as a;"'(m),m G Mn- Also, 
let An ■— U,„g^„;B„(i;"(m), -\/Z)) where S„(c",r) is the 
n-dimensional ball centered at c" with radius r. Now, let 
7„ > be such that Vol(S„(0,7„)) Vol(y^„). Clearly, 
Vol(AO < \Mn\yol{Bn{0,VD)). Since i?„ = ilog|A^„|, 



> 



Voi(A^) 



Yo\{BniO,VD)) Vol(S„(0,VD)) 



Vol(6„(0,7„)) _ ^ 7„ 
D 



Hence, we have R{a^,D) + e„ > i?„ > ^ log i.e.. 



7„ < a^e^' 



(36) 



The probability of excess distortion can be lower bounded as: 

e{fn,ipn,<y^D) = P(X" ^ An) > P(X" ^ S„(0,7„))- 

Now define the random variables Yi Xf /a"^ and note that 
the li's are -distributed. With this notation, and using ( l36b . 



e(/n,<y5n,CT^,L')>I 



Recall that for the Xi -distribution, the cumulant generating 
function is A(6') = — ilog(l — 29) and the rate function is 
I{y) — max9{6'y — A(^)} — — 1) — i log y. Furthermore, 
0*{y) :— 5(1 — I ) is the maximizer. Using the standard change 
of measure technique for the lower bound in Cramer's theorem 
(see proof of [11, Theorem 2.2.3]), 



e(/„,<^„,a ,D) > /3„exp ~nl{e - -(1 - e 



-2e„^ 



where /3„ :^ PC^Eili^^ ^ {e^'-.e^^- + t„)) and t„ is a 
sequence to be chosen. The random variables Yi have (tilted) 
distribution q{y) := exp[0*(e2'^")y — A(6'*(e2'^"))]p(y) where 



p( • ) is the Xi distribution of the Yi's. By the choice of q{ ■ 
Eq[Yi\ = e^"". Put r„ := for some C > 0. Then, 



l-/3n < 



1 

< - 
- 2 



32 



32 



where in the second inequality, we applied the Berry-Esseen 
theorem to the first term (the third moment of % is ISe"^*^") 
and Chebyshev's inequality to the second. By (|30] |. ~> | 
from below. With this choice of r„, for n sufficiently large, 

e(/„, ^n, ^^D)>^ exp [-nelil + ( + o(l))] , (37) 

where applied the facts /(e^"^") e^ + o(e^J and 1 — e^^^" = 
2e„ + o(e„). The converse in ( |32] i follows by taking the 
logarithm, normalizing by ne^, taking n — > cx), and finally 
taking ( ^ 0. ■ 
The MD setting for the probability of correct decoding of 
Gaussian sources can also analyzed analogously: 

Theorem 6. Let e„ be any positive sequence satisfying ( I30l l. 
There exists a rate-distortion code {(/„, (p„)}„gN with rates 
^ log II /nil > R{cr^,D) - e„ such that 



lim inf ■ 



n->oo ne; 



■log(l-e(/„,(^„,a2,i?)) > -1. 



(38) 



Furthermore, every rate- distortion code {{fmVn)}n&i with 
rates ^ log ||/n|| > ^(c^, — wMif satisfy 



1 



(39) 



limsup — jlog(l - e(/„,(y3„,cr ,£>))< -1. 

Proof: Similar to Theorem |5] and uses ideas in 131. ■ 

V. Conclusion 

In this paper, we analyzed the MD regime for lossy source 
coding. In analogy to Q, we showed for discrete sources that 



1 



lim — 2" log e(/n, Vn, P, D) = 



1 



(40) 



2V{P, D) 

and for Gaussian sources the RHS of (l40b is equal to —1 
independent of the variance and the distortion level D. 
As in [8J-I.10J, this reveals that the fundamental nature of the 
dispersion in the lossy source coding context. There are at 
least three avenues for future research: (i) Can the results be 
applied to, for instance, general sources as in |4|? (ii) Can 
similar analysis of the MD setting be applied to lossy source 
coding problems with side information, e.g., the Wyner-Ziv 
problem? (iii) What is the exact relationship between the MD 
and CLT regimes cf. |13|? 

Appendix: Proof of Lemma[3] 

Proof: The rate-distortion function is uniformly continu- 
ous. Specifically, R{Q, D)-R{P, D) = 0(||Q-P|| i log \\Q- 
P\\i) ED. Also, minQep„(^)||Q- P||i < \X\ln for any 
P e V{X) lUI] Lemma 2.1.2] so m.inQ(,-p^(x)R{Q,D) ~ 
R{P^D) = 0(i^^) which is asymptotically dominated by 



e'^ — Thus, there exist n-types in the regular- 

closed set "{Q £ V{X) : R{Q,D) - R{RD) > e'„} 
for n large. Let Marton's exponent be £)(Q^''||P) = 
F{P, R{P, D) + e;, D). Then, notice that 



£)(Q(")||P) i?(Q(")||P)-i?(Q^^||P 



D{Q^M^\\P) 



DiQ'u'WP) 



(41) 



The numerator of the first term on the RHS in ( 1411 1 is O(-) 



because \D{Q(^) \\ P) - Z?(Q{^'^ || P)| = 0(|ig(") - Q^^^Hi) 
and HQ'-"-' — Q^"''||i = 0{-). FromLemma|2l the denominator 
(Marton's exponent) scales as e'^ /{2V{P, D)) = 
Thus, the first term in dTTT l tends to zero and the ratio of the 
divergence in ( |23] | and Marton's exponent tends to one. ■ 
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